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Program Description^ 

The Talent Development Middle Grades Program is a comprehen- 
sive reform modei that transforms the structure and curriculum 
of iarge urban middie schoois with the aim of improving student 
achievement and raising teacher and student expectations. Key 
features of the Talent Development Middle Grades Program include 
small learning communities, an evidence-based curriculum tied 
to standards, the use of teacher teams, professional development 
and support for teachers, and school-family-community connec- 
tions. This review focuses on the effects of the Talent Development 
Middle Grades Program on student reading outcomes. Student 
Team Literature is the literacy component of the Talent Development 
Middle Grades Program, and is therefore relevant to this review in 
the Adolescent Literacy topic area. 

Student Team Literature is a reading and English language arts cur- 
riculum for middle school students that utilizes cooperative learning 
activities, high-interest reading materials, and explicit instruction to 
teach reading strategies, comprehension skills, and fluency in read- 
ing and writing. It offers integrated reading, writing, and language arts instruction and a writing process approach 
to language arts. As part of the program, students work in groups, and activities follow a regular cycle that involves 
teacher presentation, team practice, independent practice, peer preassessment, and individual assessments that 
form the basis for team scores. The cooperative learning teams are intended to engage students in academic inter- 
actions and create a motivating environment. 

Research^ 

The What Works Clearinghouse (WWC) identified one study of the Talent Development Middle Grades Program that 
both falls within the scope of the Adolescent Literacy topic area and meets WWC evidence standards with reserva- 
tions. The study included adolescent readers from grades 7 and 8 in 29 urban middle schools in the Northeast. 

The WWC considers the extent of evidence for the Talent Development Middle Grades Program on the reading 
performance of adolescent readers to be small for one outcome domain: comprehension.^ 
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Effectiveness 

The Talent Development Middle Grades Program was found to have potentially positive effects on comprehension 
for adolescent readers. Three other domains in this topic area are not covered in this intervention report. (See the 
Effectiveness Summary on p. 6 for further description of all domains.) 

Tabie 1. Summary of findings^ 


Improvement index (percentile points) 


Outcome domain 

Rating of effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Comprehension 

Potentiaily positive 

+3 

-3 to +7 

1 

nr 

Smaii 

nr = not reported 
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Program Information 

Background 

The Talent Development Middle Grades Program was developed at Johns Hopkins University by researchers, 
educators, and curriculum writers in collaboration with middle school practitioners. For more information, contact 
Kathy Nelson, Director of Implementation, Talent Development Middle Schools Program, Center for Social Organi- 
zation of Schools, Johns Hopkins University. Address: 3003 N. Charles St., Suite 200, Baltimore, MD 21218. Email: 
knelson@csos.jhu.edu. Telephone: (410) 516-6431. Fax: (410) 516-8890. 

Program details 

Key features of the Talent Development Middle Grades Program include small learning communities, an evidence- 
based curriculum tied to standards, the use of teacher teams, professional development and support for teachers, 
and school-family-community connections. Schools are systematically reorganized into small learning communities 
with interdisciplinary teacher teams. Each team shares the same students and has common planning time. In addi- 
tion to Student Team Literature, curricula are also available in mathematics, science, and US history. 

The Talent Development Middle Grades Program includes time for career exploration and courses in study skills 
and social skills. Teachers receive professional development on the use of the curriculum and accompanying 
Instructional practice. Each school employs the services of curriculum coaches to provide ongoing teacher support. 
The program also provides catch-up opportunities during the school day for students who are struggling with 
mathematics or reading. The model has been implemented with culturally and linguistically diverse populations. 

Student Team Literature^ is part of the Talent Development Middle Grades Program. Student Team Literature has 
two principal elements: literature-related activities and direct instruction in reading comprehension. First, teachers 
introduce novels to the class and discuss relevant background knowledge, genre, and vocabulary. The program 
pairs Talent Development Student Team Literature discussion guides with high-interest trade books. Discussion 
guides are available for nearly 200 works. Including over 60 award-winning novels and plays. 

Teachers begin by introducing students to the text, and then organize students into learning teams of four or five for 
reading activities. They receive rewards for working well both as individuals and as group members. Students take 
part in the following literature-related activities: 

• Partner Reading— students first read silently, then take turns reading orally with a partner. 

• Treasure Hunts— questions guide student reading, requiring them to search and think to generate text- 
supported answers. 

• Word Mastery— students practice saying new vocabulary words with their partners. They then use those 
words in story-related writing. 

• Story-Retelling— students summarize stories in their own words. 

• Story-Related Writing— students write in response to prompts about their reading. 

• Extension Activities— students conduct cross-curricular research and complete activities related to fine arts, 
dramatics, and media as they explore themes In the storles/books. 

• Tests— students take tests on comprehension, word meaning, and word pronunciation. 

A direct instruction component of the program is designed to teach students how to identify main ideas and 
themes, draw conclusions, make predictions, and understand figurative language. 
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Cost 

The cost of implementing the Talent Development Middle Grades Program is available from the developer. The 
Student Team Literature component of the program costs approximately $400 per classroom. The curricular materials 
include teacher guides, partner discussion guides for students, literature tests, and word mastery tests for novels 
and plays. Teacher training begins with a two-day summer session, followed by monthly training throughout the 
year. Teachers in the program participate in monthly seminars to troubleshoot problems with instruction, extend 
their knowledge of the program’s support materials, and enhance their skills in the program’s instructional strategies. 
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Research Summary 


The WWC identified 18 studies that investigated the effects of 
the Talent Development Middle Grades Program on the reading 
achievement of adolescent readers. 

The WWC reviewed ali of those studies against group design evidence 
standards. No studies are randomized controlled trials that meet 
WWC evidence standards without reservations, and one study 
(Heriihy & Kemple, 2004) is a quasi-experimentai design that meets 
WWC evidence standards with reservations. This study is summarized 
in this report. Three studies do not meet WWC evidence standards. 
The remaining 14 studies do not meet WWC eiigibility screens for 
review in this topic area. Citations for all 18 studies are in the Refer- 
ences section, which begins on p. 7. 


Table 2. Scope of reviewed research 


Grade 

7,8 

Delivery method 

Whole class 

Program type 

Curriculum 

Studies reviewed 

18 studies 

Group design studies 
that meet WWC 
evidence standards 

• without reservations 

• with reservations 

0 studies 

1 study 


Summary of studies meeting WWC evidence standards without reservations 

No studies of the Talent Development Middle Grades Program meet WWC evidence standards without reservations. 


Summary of study meeting WWC evidence standards with reservations 

Heriihy and Kempie (2004) conducted a quasi-experiment that examined the effects of the Talent Development Middle 
Grades Program on students in grades 7 and 8 attending middie schoois in an urban schooi district in the Northeast. 
Eleven intervention schools implemented the Talent Development Middle Grades comprehensive reform modei. 
Student Team Literature was taught as the reading and Engiish ianguage arts curricuium. Eighteen comparison 
schoois implemented the district’s standard curriculum. Participating intervention and comparison schools were 
matched on school performance, ethnic composition, English proficiency, poverty, and student mobility, and each 
intervention school was matched with a set of comparison schools. This resulted in clusters of 1-12 comparison 
schools matched to each intervention school, with some comparison schools being matched to more than one 
intervention school. Of the 1 1 Intervention schools, six used the program from 1997-98 to 2001-02, and five used 
the program in 2001-02. The authors refer to the first group as “early-implementing schools” and the latter group 
as “later-implementing schools”; we follow that naming convention in this report. An effect was measured after 
1, 2, 3, 4, and 5 years of intervention implementation. Findings that reflect maximum exposure to the intervention 
by students were used to determine the rating of effectiveness in this WWC report.® 
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Effectiveness Summary 

The WWC review of the Talent Development Middle Grades Program for the Adolescent Literacy topic includes 
student outcomes in four domains: alphabetics, reading fluency, comprehension, and general literacy. The one 
study of the Talent Development Middle Grades Program that meets WWC evidence standards reported findings in 
one of the four domains: comprehension. The findings below present the authors’ estimates and WWC-calculated 
estimates of the size and statistical significance of the effects of the Talent Development Middle Grades Program 
on adolescent readers. For a more detailed description of the rating of effectiveness and extent of evidence criteria, 
see the WWC Rating Criteria on p. 16. 

Summary of effectiveness for the comprehension domain 

One study reported findings in the comprehension domain. 

Herlihy and Kemple (2004) found, and the WWC confirmed, a statistically significant positive effect of the Talent 
Development Middle Grades Program on the State Standards Assessment (SSA) in reading for eighth-grade 
students in year 2 in early-implementing treatment schools. Herlihy and Kemple (2004) did not find statistically 
significant positive effects of the Talent Development Middle Grades Program on the SSA in reading for eighth- 
grade students in years 1,3,4, and 5 or on the Stanford Achievement Test, 9th Edition, for year 5 seventh- 
grade students in early-implementing treatment schools. For later-implementing treatment schools in year 1 , the 
authors did not find statistically significant positive effects of the Talent Development Middle Grades Program on 
the SSA in reading for eighth-grade students or on the Stanford Achievement Test, 9th Edition, for seventh-grade 
students (at p < 0.05). Because the effect on the SSA in reading for eighth-grade students in year 2 in early- 
implementing treatment schools is positive and statistically significant, and no effects are negative and statistically 
significant, the WWC characterizes these study findings as a statistically significant positive effect. 

Thus, for the comprehension domain, one study showed statistically significant positive effects. This results in a 
domain rating of potentially positive effects, with a small extent of evidence. 


Table 3. Rating of effectiveness and extent of evidence for the comprehension domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of a positive effect with 
no overriding contrary evidence. 

In the one study that reported tindings, the estimated impact ot the intervention on outcomes in the comprehension 
domain ms a statistically significant positive etfect. 

Extent of evidence 

Criteria met 

Small 

One study that included an unspecified number of students in 29 schools reported evidence of effectiveness in 
the comprehension domain. 
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Appendix A: Research detaiis for Heriihy & Kempie, 2004 

Herlihy, C. M., & Kempie, J. J. (2004). The Talent Development Middle School model: Context, components, 
and initial Impacts on students’ performance and attendance. New York: MDRC. 


Table A. Summary of findings 


Meets WWC evidence standards with reservations 

Outcome domain 

Sample size 

study findings 

Average improvement index 

(percentile points) Statistically significant 

Comprehension 

29 schools 

+3 Yes 


Setting The study took place in 29 middle schools in an urban school district in the Northeast. 

Study sample Eleven Talent Development (TD) schools and 18 comparison schools participated in this 

quasi-experimental study. The TD schools used the TD Student Team Literature curriculum for 
reading and English language arts.® Each TD school was matched with a set of non-TD com- 
parison schools that were similar on several dimensions, including racial/ethnic composition 
and math and reading test scores of eighth graders averaged over the 1995-96 and 1996-97 
school years. This matching process resulted in groups (clusters) of 1-12 non-7D comparison 
schools for each TD school, with some non-fD schools serving as comparison schools for 
more than one TD school. 

To estimate program impacts, two interrupted time series analyses were performed. The first 
compared the change in student test scores in TD schools after the program’s implemen- 
tation with the change of test scores of similar students in the same schools prior to TD’s 
implementation. The second interrupted time series analysis was conducted for the matched 
non-TD schools. Comparing the change in student test scores in TD schools to the change 
in student test scores in the matched non-TD schools produced program impact estimates. 
Specifically, the difference between deviations from the baseline in the TD schools and the 
deviations from the baseline in the non-TD schools on reading outcomes serves as the esti- 
mated program impact. 

For the analysis of early-implementing TD schools (featured in Appendices C and D of this 
report), sample sizes were from two to six intervention schools and 18 comparison schools. 

For the analysis of later-implementing TD schools, sample sizes were five intervention schools 
and 18 comparison schools. 
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Intervention 

group 


Comparison 

group 

Outcomes and 
measurement 


Support for 
impiementation 


The Talent Development Middle Grades Program includes a systematic reorganization of each 
school into small learning communities, organized around interdiscipiinary teacher teams 
that share the same students and have common pianning time. Teachers receive professionai 
deveiopment on the use of the curriculum and accompanying instructional practice, and each 
school employs the services of curriculum coaches to help support teachers on an ongoing 
basis. The model provides catch-up opportunities during the school day for students who are 
struggling with mathematics or reading. Finally, the model facilitates school-family-community 
partnerships. The TD schools used TD Student Team Literature as their reading, English, and 
language arts curriculum, with most schools adopting it, at least partially, in the first year. The 
program includes partner discussion guides to assist students as they study fiction and non- 
fiction books and work in cooperative teams. 

The early-implementing treatment schools used the TD model for five school years, from 
1997-98 to 2001-02. The start of TD implementation was staggered across the six early- 
implementing schools, with some schools beginning TD implementation earlier than others. 

All six of the early-implementing schools had at least three years of implementation experience 
when the analysis was conducted: two schools had three years of experience, two schools 
had four years of experience, and two had implemented TD for five years. The later-implementing 
treatment schools used the TD model for one school year (2001-02, the most recent school 
year for which data are available in the analysis). 

Students in the comparison group received the district’s standard reading/English language 
arts curriculum. The name of the standard district curriculum was not specified. 

The primary reading tests, administered annually in the school district, were the State Standards 
Assessment (SSA) and the Stanford Achievement Test, 9th Edition (SAT-9). The SSA was given 
in eighth grade, and the SAT-9 was given in seventh grade. Results for both tests were presented 
as Normal Curve Equivalent scores. For a more detailed description of these outcome measures, 
see Appendix B. The outcome measures that reflect the student’s maximum exposure to the 
intervention are used to determine the WWC effectiveness rating and, therefore, are reported 
in Appendix C. The intermediate findings are reported in Appendix D. 

The Talent Development Middle Grades Program provides four tiers of continuous support for 
teachers, including: (1) subject-specific professional development with a focus on modeling 
lessons, content knowledge, instructional strategies, and classroom management; (2) in-classroom 
support from a curriculum coach; (3) in-school support from teachers who receive extra training; 
and (4) support from instructional facilitators based at the Center for Research on the Education 
of Students Placed at Risk (CRESPAR). 
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Appendix B: Outcome measures for the comprehension domain 


Comprehension 


Comprehension 


Stanford Achievement Test, 
9th Edition (SAT-9) 

The SAT-9 is a standardized test used to measure academic knowledge of elementary and secondary school 
students in the United States. In this test, students answer multiple-choice questions on two reading subtests, 
reading vocabulary and reading comprehension. The scores from these two subtests were aggregated into a 
single total reading score. This test was given in seventh grade in the district, and results were presented as 
Normal Curve Equivalent scores in the study (as cited in Herlihy & Kemple, 2004). 

State Standards Assessment 
(SSA)-Reading 

This state standards assessment is a criterion-referenced test administered in grades 5, 8, and 11, which provides 
information about student performance on skills and content knowledge specified by the state. Results for this 
reading test were presented as Normal Curve Equivalent scores in the study (as cited in Herlihy & Kemple, 2004). 
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Appendix C: Findings inciuded in the rating for the comprehension domain 





Mean 








(standard deviation) 

WWC calculations 



Study 

Sample 

Intervention 

Comparison 

Mean 

Effect 

Improvement 


Outcome measure 

sample 

size 

group 

group 

difference 

size 

index 

p-value 

Heriihy & Kempie, 2004^ 

Early-implementing schools 

Three years of intervention 

SSA-Reading 

Grade 8 

6 clusters/ 

29,10 

29.00 

0,10 

0.01 

0 

>0.05 


Years 
Cohort 3 

24 schools 

(nr) 

(nr) 





SSA-Reading 

Grade 8 

4 clusters/ 

30.80 

30.10 

0.70 

0,05 

-f2 

>0.05 


Year 4 
Cohort 4 

nr schools 

(nr) 

(nr) 





SSA-Reading 

Grade 8 

2 clusters/ 

35.50 

32.70 

2.80 

0.17 

+7 

>0.05 


Years 
Cohort 5 

nr schools 

(nr) 

(nr) 





Two years of intervention 

SSA-Reading 

Grade 8 

6 clusters/ 

30.50 

27.80 

2.70 

0.17 

+7 

<0.05 


Years 
Cohort 2 

24 schools 

(nr) 

(nr) 





SAT-9 

Grade 7 

2 clusters/ 

41.60 

41,50 

0.10 

0.01 

0 

>0.05 


Years 
Cohort 6 

nr schools 

(nr) 

(nr) 





One year of intervention 

SSA-Reading 

Grade 8 

6 clusters/ 

27.70 

28.80 

-1.10 

-0.07 

-3 

>0.05 


Year 1 
Cohort 1 

24 schools 

(nr) 

(nr) 





Later-implementing schools 

One year of intervention 

SSA-Reading 

Grade 8 

5 clusters/ 

32.40 

31.70 

2.70 

0.05 

+2 

>0.05 


Year 1 
Cohort 7 

23 schools 

(nr) 

(nr) 





SAT-9 

Grade 7 

5 clusters/ 

40.70 

37.70 

3.00 

0.17 

+7 

>0.05 


Year 1 
Cohort 8 

23 schools 

(nr) 

(nr) 





Domain average for comprehension (Heriihy & Kempie, 2004) 



0.07 

+3 

Statistically 









significant*’ 


Table Notes: For mean difference, effect size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors 
the comparison group. Mean difference corresponds to the author-reported impact. The effect size is a standardized measure of the effect of an intervention on student outcomes, 
representing the average change expected for aii students who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an 
aiternate presentation of the effect size, refiecting the change in an average student's percentiie rank that can be expected if the student is given the intervention. The WWC- 
computed average effect size is a simpie average rounded to two decimai piaces; the average improvement index is caicuiated from the average effect size. The statisticai 
significance of the study’s domain average was determined by the WWC. Years 1 to 5 are reiative to the first caiendar year of Talent Development \mp\ementation in each schooi 
(Heriihy & Kempie, 2004, p. 18). Cohorts 1-8 refer to non-overiapping student sampies that received the Talent Development Middle Grades Program. nr=not reported. SSA = State 
Standards Assessment. SAT-9 = Stanford Achievement Test, 9th Edition. 

“ For Fleriihy and Kempie (2004), no correction for ciustering or muitipie comparisons was needed. The p-vaiues and effect sizes presented here were reported in the originai study. 

The comparison group vaiues are regression-adjusted estimates, which come from a muiti-step process using ordinary ieast squares and controiiing for fourth-grade math and 
reading SAT-9 test scores, race, and whether the student had repeated a prior grade. The intervention group vaiues are comparison group vaiues pius program impact caicuiated for a 
given year as the differences in deviation from the baseiine average between intervention schoois and comparison schoois. For grade 8, the effect size was caicuiated by dividing the 
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impact in a given year by the standard deviation of that outcome for aii eighth-grade students in the 1 8 comparison schoois from preintervention schooi years 1 995-96 and 1 996-97. 
For grade 7, the effect size was caicuiated by dividing the impact in a given year by the standard deviation of that outcome for aii seventh-grade students in the 1 8 comparison 
schoois from preintervention schooi years 1995-96 and 1996-97. 

This study is characterized as having a statisticaiiy significant positive effect because the effect for at ieast one measure within the domain (SSA, grade 8, year 2) is positive and 
statisticaiiy significant, and no effects are negative and statisticaiiy significant. For more information, piease refer to the I/VIVC Standards and Procedures Handbook, version 2.1 , p. 96. 
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Appendix D: Summary of additionai intermediate findings for the comprehension domain 


Mean 

(standard deviation) WWC calculations 



Study 

Sample 

Intervention 

Comparison 

Mean 

Effect 

Improvement 


Outcome measure 

sample 

size 

group 

group 

difference 

size 

Index 

p-value 

Herlihy & Kemple, 2004^ 

Early-implementing schools 

Two years of intervention 

SAT-9 

Grade 7 

6 clusters/ 

35.20 

35.70 

-0.50 

-0.03 

-1 

>0.05 


Year 2 
Cohort 3 

24 schools 

(nr) 

(nr) 





SAT-9 

Grade 7 

5 clusters/ 

37.20 

36.20 

1.00 

0.06 

-f2 

>0.05 


Years 
Cohort 4 

23 schools 

(nr) 

(nr) 





SAT-9 

Grade 7 

4 clusters/ 

37.50 

35.90 

1.60 

0.09 

-f4 

>0.05 


Year 4 
Cohort 5 

nr schools 

(nr) 

(nr) 





One year of intervention 

SAT-9 

Grade 7 

6 clusters/ 

36.20 

36.20 

0.00 

0.00 

0 

>0.05 


Year 1 

24 schools 

(nr) 

(nr) 






Cohort 2 


Table Notes: The supplemental findings presented in this tabie are additionai findings from the study in this report that do not factor into the determination of the effectiveness rat- 
ing for the intervention. For mean difference, effect size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative num- 
ber favors the comparison group. Mean difference corresponds to the author-reported impact. The effect size is a standardized measure of the effect of an intervention on student 
outcomes, representing the average change expected for aii students who are given the intervention (measured in standard deviations of the outcome measure). The improvement 
index is an aiternate presentation of the effect size, reflecting the change in an average student’s percentile rank that can be expected if the student is given the intervention. Years 
f to 4 are relative to the first calendar year of Ta/enf Deve/opmenf implementation in each school (Herlihy & Kemple, 2004, p. f 8). Cohorts 2-5 refer to non-overlapping student 
samples that received the Talent Development Middle Grades Program, nr = not reported. SAT-9 = Stanford Achievement Test, 9th Edition. 

For Flerlihy and Kemple (2004), no correction for clustering or multiple comparisons was needed. The p-values and effect sizes presented here were reported in the original study. The 
comparison group values are regression-adjusted estimates, which come from a multi-step process using ordinary least squares and controlling for fourth-grade math and reading SAT-9 
test scores, race, and whether the student had repeated a prior grade. The intervention group values are comparison group values plus program impact calculated for a given year as the 
differences in deviation from the baseline average between intervention schools and comparison schools. For grade 7, the effect size was calculated by dividing the impact in a given year 
by the standard deviation of that outcome for aii seventh-grade students in the 1 8 comparison schools from preintervention school years 1 995-96 and 1 996-97. 
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Endnotes 

■' The descriptive information for this program was obtained from publicly available sources: the program’s website (http://www. 
csos.jhu.edu, downloaded July 2010) and the research literature (Mac Iver, Plank, & Balfanz, 1997). The WWC requests developers 
review the program description sections for accuraoy from their perspective. The program desoription was provided to the devel- 
oper in October 201 1 ; however, the WWC received no response. Further verification of the accuracy of the desoriptive information 
for this program is beyond the scope of this review. The literature search reflects documents publicly available by December 201 1 . 

^ The studies in this report were reviewed using the Evidence Standards from the WWC Procedures and Standards Handbook (version 2.1), 
along with those described in the Adolescent Literacy protocol (version 2.1). The evidence presented in this report is based on avail- 
able research. Findings and conclusions may change as new research becomes available. 

® The study that meets WWC evidenoe standards with reservations did not examine the effectiveness of the Talent Development 
Middle Grades Program on adolesoent readers in the alphabetics, reading fluency, or general literacy achievement domains. 

^ For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 1 6. These 
improvement index numbers show the average and range of student-level improvement indices for all findings across the studies. 

® The Student Team Literature program is an adaptation and elaboration of the Student Team Reading and Writing program (see WWC 
intervention report released in December 2011). Compared to Student Team Reading and Writing, Student Team Literature focuses on 
books rather than short seleotions from an anthology and includes a staff development component. 

® The Talent Development Middle Grades Program was implemented in grades 6-8. The findings considered for the effectiveness rating 
reflect the maximum exposure of students to the program. For example, in the third, fourth, and fifth years of Talent Development 
implementation, eighth graders (from Cohorts 3, 4, and 5) had been exposed to the program over a period of three years (when they 
were in the sixth, seventh, and eighth grades). The corresponding intermediate findings (after two years of implementation) for seventh 
graders from the same Cohorts 3, 4, and 5 are reported in Appendix D and were not used for the rating of effectiveness. For Cohorts 2 
and 6, two-year findings are considered for the effectiveness rating because these findings reflect the maximum exposure to the read- 
ing program for those students. For later-implementing schools (Cohorts 7 and 8), one-year findings (for seventh and eighth graders) are 
considered for the effectiveness rating because these findings reflect the maximum exposure to the reading program for those sohools. 

^ Follow-up findings reported in Herlihy and Kemple (2005) are not included in this report because baseline equivalence was not 
demonstrated between intervention and comparison groups. 

® One early-implementing school did not fully adopt Student Team Literature in all classrooms and eventually dropped it in the fifth 
year of Talent Development implementation. One of the later-implementing treatment schools never adopted the Student Team 
Literature program. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, January). 
Adolescent Literacy intervention report: Taient Deveiopment Middie Grades Program. Retrieved from 
http://whatworks.ed.gov. 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 


Study rating 

Criteria 

Meets WWC evidence standards 
without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a weii-implemented RCT. 

Meets WWC evidence standards 
with reservations 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high attri- 
tion that has established equivaience of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC evidence stan- 
dards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of sfudies show indeterminafe effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show a statisti- 
cally significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students in a 
class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Single-case design 
Standard deviation 


Statistical significance 


Substantively important 


Attrition occurs when an outcome variable is not avaiiabie for aii participants initiaiiy assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 16. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 1 6. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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